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ABSTRACT 



An operator interface for a video editing system pro- 
vides a visual sense of the content of video sequences, as 
well as their length, while also providing enhanced 
interactive control of locations and time alignments of 
the video. As the video sequence is processed into the 
system, a small but representative sample of each frame 
is saved in a local memory, while the video itself is 
stored in mass storage. These samples are used to pro- 
vide a video pictorial timeline of the underlying stored 
video. The location of an operator's view into the video ' 
sequence is controlled by a cursor's movement along a 
detailed video pictorial timeline, a reverse motion area 
and a forward motion area to provide VTR control for 
location changes on the video tape. The cursor's move- 
ment can be controlled by a mouse or a knob. Icons, 
either static or dynamic, are produced within the mo- 
tion areas to indicate the amount of selected velocity. 
Timelines can be marked with time marks, roughly 
aligned and then automatically fine aligned by the sys- 
tem according to their respective time markers. The 
editing results associated with these timelines are also 
time aligned as a result of this process. 



10 Claims, 13 Drawing Sheets 
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quences as they are being manipulated, but these simple 

. VIDEO EDITING SYSTEM OPERATOR timelines do not provide any visual cues as to the con- 

INTER-FACE FOR VISUALIZATION AND tents of the video sequences 

INTERACTIVE CONTROL OF VIDEO MATERIAL i a connection with another aspect of the prior art, 

^ j * .-^o^t 5 control of the displayed location on a video tape in a 

07^Q 3 ? 7 LH^TSqi h PP k D N °* te P e recorder traditionally been accomplished 

07/739,357 filed Aug. 2, 1991 and now abandoned. ^ tWQ wheds Qr ^ qt a ^ ^ 

BACKGROUND OF THE INVENTION which operates in two modes, depending on the posi- 
This invention relates to video editing, and more 10 ^ of&m °te c °**°* switch. In the first modeofoper- 
particularly to an operator interface for a video editing atl0n * r0tatl0Iial motlon of ^ whccl or » inter- 
system that provides enhanced visualization and inter- P ret ed as positional distance. In the second mode of 
active control of video sequences during the editing operation, the same rotational motion is interpreted as 
process. velocity magnitude control. U.S. Pat. No. 4,988,982 to 
Presently, the video editing equipment that is used 15 ^ Present inventor for "Touch Pad Machine Con- 
tends to interpose a lot of numbers and technical consid- trol", hereby incorporated by reference, describes this 
erations between an artist attempting to create a video prior art in its "Background of the Invention". The 
program and the video images that must be manipulated specification of the U.S. Pat. No. 4,988,982 describes an 
in the course of creating that program. Cutting and alternative approach to VTR control that relies on a 
splicing film on a film editor provided a visual and 20 touch pad with different regions, one of which is de- 
spatial dimension to the temporal aspect of a film. Thus voted to position control, another of which is devoted 
far, efforts to bring this same sort of interactive conve- to velocity control, 
nience and "feel" to the video tape editing process have 

met with limited success, although considerable SUMMARY OF THE INVENTION 

progress has been made 25 it is an object of the present invention to provide an 

U.S. Pat No. 4,937,685 to Barker et al for a "Method oper ator interface for a video editing system that dis- 

ofDisplayPr^taUonfor Video Editing" discloses an plays a representation of a £ d eo sequence to 

videochp^ideolabels^^ 30 "TS^^^^r t* ■ * 

sentations of a frame of actual video. Because these According to one aspect of the invention, as the 

video labels contain much less total information than ? ade0 se<luence 13 P rocessed mto svstem > a sma " 

the underlying video frame that they represent, they ? ut re P rMenta tive sample of each video frame is saved 

can be more readily stored and retrieved than actual m a local memorv » whiIe the video itself is stored in 

video images can be, and are therefore used as surro- 35 mass stora £ e - ^ e frame samples are used in the display 

gates for the real video images during some parts of the to P rovi de a video pictorial timeline of the underlying 

editing process. stored video. 

In the system described in the Barker et al '685 patent, It is a further object of the present invention to pro- 

the operator can view a clip of video or a transition vide an operator interface for a video editing system 

between two clips of video on a main monitor in an 40 that allows interactive and efficient positioning within 

"endless loop" repeating display. A set of four label the video sequence, 

display screens show the above described video labels According to another aspect of the invention, the 

of the clips currently under active manipulation. A location of an operator's view into the video sequence is 

plurality of additional smaller display monitors are pro- controlled by a cursor in a machine control window, 

vided in vertical pairs for viewing a series of pairs of 45 The machine control window contains a detailed video 

video labels representing a series of clips currently ar- pictorial timeline, a reverse motion area and a forward 

^l^w^^f^f^Vu ♦ • ui motion area. The interaction of the cursor with the 

IntheBarkeretal 685 system, the operator -is able to reverse motion w selected ^ 

of ^e video itself, verse velochy d ^ ^ ce seen fa ^ 

stored* its orW^ S^ly. the mte^ 

fast access to the video itself. When rapid access to the f 10 *™*** f °*kcs ****** founts of forward ve- 
actual video is required, it is achieved by recording ^ng the video sequence. Icoiis, which can either 
multiple copies of the same video clip on multiple video 55 be static or dvnamic > produced within the motion 
tape recorders (VTRs) and positioning each of the io mdicate amount of selected velocity. The 
VTRs at different locations on the clip, so that one curs °r can be controlled by either a mouse or a knob. 
VTR is always close to any desired point of interest It is a further object of the present invention to pro- 
While the Barker et al '685 system provides the oper- a met hod of video tape recorder position and ve- 
ator with video labels at the beginning and end of clips, 60 joc^y control that makes such control easier and more 
a sequential continuous loop display of the active clip intuitive for the operator. 

and a readout of the duration of the active clip, it does According to another aspect of the invention, a single 

not provide any graphic display of the clip's length or k^ 00 na s three regions, one for tape position control 

the rest of its internal contents. and one each for forward and reverse tape velocity 

A video editing software system called "Media 65 control. Rotational motion of the knob into the latter 

Maker" is produced by Macromind Inc., 410 Townsend two regions produces velocity changes, with the cur- 

St, Suite 408, San Francisco, Calif. This system displays rent velocity being determined according to the follow- 

simple timelines that mdicate the length of video se- ing relationship: 
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is a Workspace 15 that has a vertical layers axis 25 along 

_ D nr its left side and a horizontal time axis 24 along its top. 

Veur ~ Dm** * Vmax The scale on horizontal time axis is variable, allowing 

the operator to select the time span that will be dis- 

where, 5 played. 

Vcur is the current velocity. The Workspace 15 shows, for a portion of a program 

Vmax is the maximum velocity, being created, the sequential arrangement of video seg- 

Vcur is the current distance into the region, and ments or clips, as well as the relative synchronization of 

Dmax is the maximum distance into the region. those video clips (layers) that are to play at the same 

It is another object of the present invention to pro- 10 time. The Workspace 15 is divided into two parts, a 

vide an operator interface for a video editing system background area 33 on the bottom and a layers area 32 

that automatically performs fine time alignment of two 0 n top. The background area 33 is used for basic editing 

video sequences that have been marked and coarsely that only involves a linear arrangement of video sequen- 

aligned by an operator. t< ces with cuts or dissolves. The material that appears 

Accordmg to another aspect of fcemvention, two or 15 here fonn ^ background for a morc c0 ^ plex 

more timelines can be marked wtth time marks and the M ^ ^ mto h m ^ k ^32. 

video editing system then automatically performs a fine In ^ backgroim 7 a ^a 33, video segments are shown in 

time alignment of the tmehnes accordmg to their re- ^ £ ^ ^ J£ 

spective time markers. When an editing session is com- ^ J . J A . l}:< • , t ; j , ( ^J, „ 1 ' \~ 

pleted, the results of that editing sessioi reflect the time 20 * e « l^™?* ' a ^? led "Blk" and whose 

synchronization produced by this time alignment pro- Porto » ta^ B Vtr , or as a title, such as 

cess the darker shaded timeline above the "B Vtr" timeline, 

The subject matter of the present invention is particu- the ooe ^ ^ping leading edge that represents a 

larly pointed out and distinctly claimed in the conclud- transition. Audio sources associated with the wide time- 
ing portion of this specification. However, both the 25 ^ aPP^ ™ narrow timelines beneath it. 

organization and method of operation, together with 1° the layers area 32, video segments are shown as 

further advantages and objects thereof, may best be simple, narrow timelines 21. Transitions or effects in the 

understood by reference to the following detailed de- layers area 32 are shown as separate individual time- 

scription and accompanying drawings. lines. Layers in the Workspace 15 that are currently 

BRIEF DESCRTPTTONT OF THF HR A WINTfV? 30 mcluded m edit are shaded > 35 lavers U and 

BRIEF DESCRIPTION OF THE DRAWINGS L2 are, while layers that are not currently included in 

FIG. 1 is a screen view of a portion of the operator the edit are shown in white (unshaded), as layers L5 and 
interface of the present invention; L4 are. 

FIG. 2 is a simplified block diagram of the video The lower right quadrant of the screen shown in 
editing system; 35 FIG. 1 contains a Text Timeline 35 that provides a 

FIG. 3 is a block diagram representation of how a textual display of the composite timeline being devel- 
video pictorial timeline is created; 0 ped in the Workspace 15 areas 32 and 33. The lower 

FIG. 4 is a block diagram representation of how time left quadrant of the screen displays an Edit (Decision) 
marks and cursors are generated; List 3$ (EDL) of editing previously performed. The 

FIG. 5 is a block diagram representation of how a 40 Edit List 36 area can be scrolled to look at old versions 
niachme control window operates; of ^ or at a partia] resuh of ^ currcnt cdhin 

FIG. 6 illustrates an implementation of a machine cess being performed in the Workspace 15 areas 32,33. 
control window utilmng a potentiometer instead of a M work ^ comp i eted in the Workspace 15 areas 32,33, 

m0 ^f * e m Tn% °£ r ? to ' « . v a it is appended to the current Edit List 36 at the location 

FIGS. 7A and 7B illustrate the process of marking 45 of vertical re ference line 29. 
and time-ahgnmg two video sequences to achieve time a o ^ u„ 0 ™ :„ 1 ^ — ~ , c 

alignment between them- FIG- 1, the first ten seconds of an 

S^o . .„ ^ . , ^ v , edit is shown in the Workspace 15 layers area 32, the 

FIG. 8 is an illustration of three simple timelines and , nn , _ , *~ ^ y 7\ TT 

two video pictorial timelines in a single Workspace; and back^ound area 33 and the Text Timeline 35. At the 
FIGS. 9A, 9B, 9C, 9D and 9E are a simplified logic 50 ? ft " de ° f Workspace 15, five seconds of pre-roll 
flow diagram of a portion of the operator interface for for ^ « shown. Following .that ; m the 

a video editing system in accordance with the present background area 33 and m the Text Timeline 35 two 
invention. seconds of black background is shown. Audio 1 (Al) 

and Audio 2 (A2) arc also black during this time, indi- 
DETAILED DESCRIPTION 55 eating a silence on the audio tracks associated with the 

The display shown in FIG. 1 is part of a user interface black ^co. The interval of black video and silent audio 
implemented on an Apple Macintosh computer. It uti- is shown m the Text Timeline 35 as the second entry, "0 
lizes window and cursor control approaches that are BLK". The first entry in the Text Timeline 35 

. somewhat similar to the normal Macintosh (TM) inter- indicates that this recording (RECD) will use Audio 1, 
face, but instead of the normal Macintosh keyboard this 60 Audio 2 and Video (A12V), and that it will be appended 
system employs a keyboard that has colored keys. That to the Edit List 36 at time 10:00:00:00 in the Edit List 36 
keyboard resembles the keyboards and color codings frame of reference. 

that are somewhat standard in the television post-pro- The third entry in the Text Timeline 35 indicates that 
duction editing equipment industry. at time two seconds, 2:00, there will be a dissolve, 

The screen shown in FIG. 1 is divided into four quad- 65 DISS, lasting sixty frames, 60F, to the B VTR which is 
rants. The upper left quadrant contains most of the to start at time 2:00:12:03 in the B VTR frame of refer- 
features associated with the present invention and will ence. This same information is shown in the background 
be discussed in detail below. The upper right quadrant area 33 by the white box with the crossed lines as diago- 
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nals. Here the duration of the dissolve is shown with the 
notation "60f\ 

The fourth entry in the Text Timeline 35 indicates 
that at time four seconds, 4:00, another source, E VTR, 
will PLAY starting at time 1:23:05:23 in the E VTR 5 
frame of reference. This same information is shown 
visually in the layers area 32 on layer L2. 

The fifth entry in the Text Timeline 35 indicates that 
at time 4:00 the K'SCOPE (a Kaleidoscope special 
effects device) is to play. The same event is shown in 10 
the layers area 32 on the layer below layer LI, the layer 
labeled "KSc". 

The sixth entry in the Text Timeline 35 indicates that 
at time 5:15 a TITLE-A will be KEYed in over an 
interval of 10 frames, 10F. This same information is IS 
presented in the background area 33 by the dark bar 
with the diagonal leading edge that is located above the 
timeline labeled "B Vtr". The diagonal edge indicates 
that the title dissolves in, with the length of the dissolve 
being represented by the length of the diagonal portion 20 
of the bar. At the same time as the title is keyed in, 5:15, 
a new audio source, A3, is shown entering the record- 
ing by the entry RECD A123V. At time 6:00 the C 
VTR is to start playing, and at time 7:20 its MOTION 
is changed to 50% of normal playing speed. At time 25 
7:10 audio source A3 is seen to end with the entry 
RECD A12V. 

Referring now to the left side of FIG. 1, two video 
pictorial timelines 10 and 11 according to the present 
invention are visible. A main video pictorial timeline 10 30 
appears in the region that separates the upper left quad- 
rant and the lower left quadrant A more detailed video 
pictorial timeline 11 appears above the main video pic- 
torial timeline, in the lower portion of the upper left 
quadrant These video pictorial timelines provide the 35 
operator with a sense of the image presented by the 
video sequence. As will be further explained below, 
when a new video sequence is put into the video editing 
system of the present invention, a sample is taken of 
each frame. This sample is small, typically only eight 40 
pixels of information, but when these frame samples are 
assembled into a video pictorial timeline 10,11, this 
limited data provides enough information to visually 
cue the operator quite effectively. 

The main video pictorial timeline 10 shown in FIG. 1 45 
represents a video sequence that can vary in length 
from a few seconds to an hour and a half. The desired 
scaling factor is produced by duplicating or decimating 
the frame samples representing the video sequence so 
that a representative sample of the samples is displayed 50 
in the video pictorial timeline 10. Immediately above 
the main video pictorial timeline 10 there is an associ- 
ated main audio level timeline 22. As the name suggests, 
the audio level timeline 22 illustrates the audio ampli- 
tude along the main video pictorial timeline 10 by dis- 55 
playing a single-sided "envelope" indicative of the am- 
plitude level at each point in time. 

In either of the video pictorial timelines 10,11 or the 
simple timelines 21 in the Workspace 15 areas 32,33, 
clips can be taken from a longer video sequence by 60 
clicking a control button on the mouse or other cursor 
control device while the cursor 12 is on the mark In box 
26 to identify a starting point or the mark Out box 28 to 
identify an ending point. Reference marks 31 for per- 
forming time alignment between different pieces of 65 
video can be placed anywhere within a video sequence 
using the Ref Mk box 30 or a corresponding key on the 
keyboard (66 in FIG. 2). 



Video window 14 displays the video image at the 
current location within any video sequence that is acti- 
vated, including the composite video segments that are 
the result of an editing session. The video window 14 
displays the scene at the location of cursor 12 along the 
video pictorial timeline 10, whether it is still or moving, 
at normal speed or higher speeds. Immediately to the 
right of the video window 14 is a source status display 
34. Six VTRs, A through F, are listed as sources, with 
squares shown to the left of those that are stopped. 
Sources that are playing forward at normal speed, such 
as the B source, are indicated with a pointing to 
the right. A ">>" indication signifies fast forward, 
while a "<<" indication to the left, as is shown for 
source A, indicates fast rewind. Below the list of 
sources there is a RECord box 37, a PreVieW box 38 
and a palette of most used transitions 39. Clicking the 
control button on the mouse while the cursor 12 is 
positioned on the RECord box 37 causes the activity 
represented by all of the active layers in the layers area 

32 and the background area 33 to be recorded into the 
recording medium. Clicking on the PreVieW box 38 
causes a "virtual edit" of the active layers to be pre- 
viewed as a composite sequence but not recorded, i.e., 
be made visible in the video window 14 as they would 
appear if recorded. Clicking on either the "30" (frames) 
box or the "60" (frames) box in the palette of most used 
transitions 39 allows the user to drag a transition of that 
length to the location of his or her choice in the Work- 
space 15 areas. 

The content of the main video pictorial timeline 10 
changes to reflect whichever video sequence has most 
recently been identified as the current active one by the 
operator clicking the control button on the mouse 70. 
The current active video sequence can be designated 
either by clicking on its source in the status area 34 or 
on its simple timeline 21 in the Workspace 15 areas 32 or 

33 in the upper-right quadrant or by the use of keys on 
the keyboard. The main video pictorial timeline 10 can 
display active video sequences that are a single clip of 
video or, when the preview (PVW) box 38 is clicked 
on, the main video pictorial timeline 10 can display a 
composite sequence of video that has been virtually 
edited into a single sequence in one of the work areas 32 
or 33, as was explained above. 

The user's movable cursor 12 can be locked onto one 
of the cursors (also 12 in this Figure) that are perma- 
nently associated with the main video pictorial timeline 
10 and the detailed video pictorial timeline 11 by mov- 
ing the movable cursor over the associated cursor and 
clicking the control button once. The user's movable 
cursor can be unlocked from the cursor 12 associated 
with one of the video pictorial timelines 10,11 by click- 
ing the control button twice. 

While the user's movable cursor 12 is locked to one of 
the cursors associated with a video pictorial timeline 10 
or 11, moving the cursor 12 causes one of two results, 
depending on whether the mass storage means (item 50 
in FIG. 2) is linear, such as a conventional video tape 
recorder, or random access, such as laser discs or multi- 
ple hard discs. As will be further described with refer- 
ence to FIG. 5, if the system's mass storage means 50 is 
linear, moving the cursor 12 faster than the linear stor- 
age means can move the video tape causes a second 
cursor (13 in FIG. 5) to appear. The second cursor 13 
shows the actual VTR position on the tape, while the 
cursor 12 under the operator's control represents the 
desired tape position. When the cursor 12 slows down 
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or stops, the other cursor, representing the actual posi- ward motion area 20 to the right of the detailed video 
tion, moves to catch up with the desired position at the pictorial timeline 11, the image in the video window 14 
maximum speed of the linear storage means 50. If the fast forwards and the scenes represented by the frame 
system's mass storage means 50 is random access mem- samples in the detailed video pictorial timeline 11 
ory of some kind, no second cursor appears when the 5 stream to the left. 

cursor is moved rapidly and the video window 14 re- The speed of these forward and reverse motions de- 
flects the cursor 12 position almost instantaneously. pends on how far beyond the ends of the detailed video 

If the user clicks the mouse 70 button while the cur- pictorial timeline 11 the movement takes the desired 
sor 12 is on the main pictorial timeline 10, but not on the position cursor 12. When the desired position cursor 12 
cursor 12 associated with the main pictorial timeline 10, 10 reaches the outside end of either of the motion areas 
the cursor 12 associated with the main pictorial timeline 18,20, the physical tape speed is maximum in that direc- 
10 becomes a current position cursor 13 and the cursor tion. As the magnitude of this attempted movement 
12 at the position determined by the mouse 70 becomes increases, an indicator (to be further described in con- 
a desired position cursor 12. If the mass storage means nection with FIG. 5) fills the forward motion area 20 or 
50 is random access, the current position cursor will 15 the reverse motion area 18 in proportion to how fast the 
almost instantaneously move to the desired position and video sequence is moving relative to its maximum 
the image at that location will be displayed in the video speed. When this indicator fills the forward or reverse 
window 14. If the mass storage means 50 is linear, e.g., motion areas 20 or 18, the velocity of the video se- 
VTRs, the current position cursor 13 will move toward quence movement is at its maximum value, 
the position of the desired position cursor 12 at the 20 When the operator wants to slow down or stop the 
maximum speed of the VTR and the images along the motion of the video sequence through the detailed 
way win appear in the video window 14. video pictorial timeline 11, he uses the mouse to move 

The more detailed video pictorial timeline 11 directly the desired position cursor 12 back towards the detailed 
above the main video pictorial timeline 10 is part of a video pictorial timeline 11 portion of the machine con- 
"machine control window". The other parts of the 25 trol window display, thereby slowing the velocity of 
machine control window are the reverse motion area the video sequence motion. Completely returning the 
18, the forward motion area 20 and the desired position desired position cursor 12 to the detailed video pictorial 
cursor 12. The detailed video pictorial timeline 11 con- timeline 11 stops the motion of video sequence entirely, 
tains a four second interval of video, with each video Referring now to FIGS. 2 and 3, the frame samples 17 
frame sample shown two pixels wide for clarity. A 30 that make up video pictorial timelines 10 and 11 are 
machine control window audio level timeline 23 shows produced by a digital video and audio processor 58 and 
the amplitude of the audio associated with the video stored in a local memory 60 that is convenient to the 
shown in the detailed video pictorial timeline 11. A time graphic interface display generator 57. As will be fur- 
mark 31 seen above the machine control window audio ther described below, time marks 31 and single video 
level timeline 23 marks a particular location in the de- 35 frames 19 are also stored in the local memory 60. 
tailed video pictorial timeline 11 and its associated ma- The underlying video frames 19 that the frame sam- 
chine control window audio level timeline 23. pies 17 represent are stored remotely in mass storage 

The machine control window permits the operator to means 50. The video frame samples 17 are stored with 
scroll slowly through individual frames of the currently pointers to the locations of the corresponding video 
active video sequence, whether the storage is linear or 40 frames 19. These pointers, depending on the nature of 
random access (non-linear). It is used for fine control to the mass storage means 50, are either the SMPTE time 
identify and mark individual frames in the video se- codes, or memory addresses or both. While the icons 
quence with time marks 31 for use in time aligning shown for the mass storage means 50 in FIG. 2 suggest 
different video sequences. As the desired position cur- VTRs, they in fact can be any other video storage 
sor 12 is moved along the detailed video pictorial time- 45 means and are preferably high capacity random access 
line 11, its motion is reflected in the content of the video memory, such as multiple laser discs or magnetic hard 
window 14 and in the position of the cursor 12 associ- discs. 

ated with the main video pictorial timeline 10, although The artist or editor operator controls operation 
it should be noted that the motion of the latter may be through a keyboard 66 and a mouse or other cursor 
nearly imperceptible, depending on the scale factor of 50 control device 70 while interacting with the CRT dis- 
the main video pictorial timeline 10 and how that relates play 68. Keyboard commands are processed by the 
to the view provided by the detailed video pictorial human interface logic portion of block 64 which also 
timeline 11 in terms of the ratio of their time scales. contains manual controls. As will be further described 
To access a location that is beyond one of the ends of below, most of the activity associated with the present 
the detailed video pictorial timeline 11, the desired posi- 55 invention occurs within the graphic interface display 
tion cursor 12 is moved past that end. Moving beyond generator 57, the digital video and audio processor 58 
the ends of the detailed video pictorial timeline 11 and window and screen display generator 62. These 
causes the detailed video pictorial timeline 11 and its interact with local memory 60, cursor control device 70 
associated audio level timeline 23 to start moving in the and the CRT display 68 to effect most of the behaviors 
opposite direction to bring the desired location into 60 associated with the invention. 

viw- Operator input from the keyboard interacts with 

If the movement of the desired position cursor 12 is manual control human interface logic 64 to produce 
into the reverse motion area 18 to the left of the video desired effects in the editor logic and automatic control 
pictorial timeline 11, the video image in the video win- software 54. The video source interface 56 interacts 
dow goes into reverse motion as the scenes represented 65 with the editor logic and automatic control software 54 
by the frame samples in the detailed video pictorial and the manual control human interface logic 64, re- 
timeline 11 stream to the right. Conversely, if the move- ceiving instructions from them and returning status 
ment of the desired position cursor 12 is into the for- information. Responsively to these instructions, the 
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video source interface 56 controls access to the contents 
of the mass storage means 50 via selecting means for 
video sources 52. The selecting means for video sources 
52 stores (and retrieves) timecode and control data and 
video and audio information in (and from) the mass 5 
storage means 50. The video source interface provides 
identification of the video source, the current video 
frame data and the timecode range to be displayed to 
the graphic interface display generator 57. The graphic 
interface display generator 57 controls the window and 10 
screen display generator 62 to produce the display on 
the CRT display 68. 

Referring now to FIG. 3, the digital video and audio 
processor 58 produces a frame sample 17 for each video 
frame 19, and sends these frame samples to the local 15 
memory 60. In one embodiment, each frame sample 17 
that is taken consists of eight individual pixels distrib- 
uted vertically along the center of the video frame 19. 
In another embodiment, the frame sample 17 consists of 
eight composite pixels that each represent the average 20 
video values within eight different regions distributed 
vertically along the center of the video frame 19. In 
either case, these frame samples 17, when displayed in 
sequence as a video pictorial timeline, represent an 
"edge view" of the contents of the underlying video 25 
sequence. 

Other methods of producing the frame samples 17 are 
envisioned. One of these requires considerably more 
processing, but provides the operator with an even 
more intuitive feel for the activity occurring within the 30 
video sequence being represented. This approach relies 
on an algorithm to identify the most dramatic features 
of the luminance and chrominance values found in a 
frame 19 and emphasize those in the edge view repre- 
sentation. Even when these frame samples 17 are the 35 
result of sophisticated processing and communicate 
more content to the viewer, they are very small in terms 
of the data required compared to the data content of the 
video frame 19 that they represent 

Audio recordings that are input to the system, 40 
whether or not they are directly associated with a video 
recording, are processed to produce audio envelopes 
22,23 that provide a visual representation of the audio 
activity over time. The data required to reconstruct 
these audio envelopes is also stored in the local memory 45 
60. Audio is sampled 60 times per second and stored as 
eight bits of data. The audio envelope data is also stored 
with pointers to where the underlying audio is stored in 
the mass storage means 50 (FIG. 2). 

The video source interface 58 contains a pictorial 50 
timeline generator 59 and a current tape/source position 
detector 55. The video source interface 58 receives 
source identification and zoom control (time range to be 
displayed) input from the human interface logic 64 and 
timecode, video and audio information from the mass 55 
storage means 50 (both shown in FIG. 2). Based on the 
zoom control information, the pictorial timeline genera- 
tor 59 determines how much the video frame sample 
record must be decimated for display. Changing the 
zoom control alters the dynamic range of the video 60 
pictorial timeline 10 from ninety minutes to a few sec- 
onds. It also correspondingly changes the detail that can 
be seen as a result of the video frame sample decimation 
that must occur to provide room for the longer times. 

The current tape/source position detector supplies 65 
the graphic interface generator 57 and the pictorial 
timeline generator 59 with information about the cur- 
rent position on the video tape or other source. The 
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pictorial timeline generator 59 controls which video 
pictorial timeline 10,11 and audio level timeline 22,23 
data the local memory 60 is to supply to the graphic 
interface generator 57. Using the current tape/source 
position information provided by the current tape/- 
source position detector 55 and the pictorial timeline 
data supplied from the local memory, the graphic inter- 
face generator 57 operates through the window and 
screen display generator 62 to create the video pictorial 
timelines 10 and 11 and the audio level timelines 22 and 
23 on the CRT display 68 (FIG. 2). 

FIG. 4 is a block diagram representation showing 
how cursors 12,13 and time marks 31 are produced and 
applied to the video pictorial timelines 10 or 11. As 
partially described above, one cursor 12 is controlled by 
the operator to indicate the desired position on the 
video sequence, while a second cursor 13 indicates the 
actual position of a video tape recorder or other linear 
mass storage means 50 on the video tape or other me- 
dium. 

The operator controls cursor 12 via the mouse 70 or 
other cursor control device and the user's cursor gener- 
ator 72. A cursor position detector 74 monitors the 
cursor 12 position and provides timeline position com* 
parator 76 with the user's cursor 12 position. The time- 
line position comparator 76 also receives current posi- 
tion on the timeline information from the timeline cur- 
rent position location detector 61. By comparing these 
two inputs, the timeline position comparator 76 is able 
to generate position error information. A timecode 
translator 77 converts the position error information 
from the timeline position comparator 76, which is in 
the frame of reference of the CRT display (68, FIG. 2), 
into desired location changes in timecodes which are in 
the frame of reference of the stored video. These de- 
sired changes are then provided to the video source 
interface 56 which sends a corresponding motion com- 
mand to the VTR or other source. 

The video source interface 56 receives timecode and 
status information from the VTR or other source. This 
current source position information is detected by the 
current tape/source position detector 55 within the 
video source interface 56 which supplies it to the time- 
line current location detector 61 and to the graphic 
interface generator 57. The timeline current location 
detector 61 supplies current position on the timeline 
information to the timeline position comparator 76, as 
mentioned above, and to the cursor generator for 
source position 80. The cursor generator for source 
position 80 produces the current position cursor 13 on 
the CRT display 68 at the location indicated by the 
current position on timeline information. 

In addition to the current source position information 
mentioned above, the graphic interface generator 57 
also receives time display range (zoom) information and 
"Mark" command information from user input. When a 
mark command is received, the graphic interface gener- 
ator 57 produces a time mark 31 at that location. This 
time mark 31 is stored in the local memory 60 along 
with the frame samples 17 that constitute the video 
pictorial timelines 10,11 so that future video pictorial 
timelines will include this time mark 31 at this location. 
The graphic interface generator will continue to gener- 
ate a time mark 31 at the stored location until it receives 
a delete command (not shown) while the cursor posi- 
tion detector 74 is indicating the position of this old 
mark. 
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FIG. 5 is a block diagram representation that shows knob 81 as active knob position indicator 79 moves 
how the machine control window operates. A desired through a positional region 85. 
position cursor 12 is controlled using the mouse or other Sufficient counter-clockwise rotation of the control 
cursor control device (70 in FIG. 4) and the user's cur- knob 81 takes the active knob position indicator 79 out 
sor generator (72 in FIG. 4). This cursor position is 5 of the positional region 85 and into a reverse region 83, 
monitored by the cursor position detector 74. The out- while sufficient clockwise rotation of the active knob 
put of the cursor position detector 74 is monitored by a position indicator 79 takes the active knob position 
variable rate detector 86. Based on whether the desired indicator 79 out of the positional region 85 and into a 
position cursor 12 is in the reverse motion area 18 or the forward region 87. Maximum counter-clockwise rota- 
forward motion area 20 and how far into one of those 1° tion of the control knob 81 causes the desired position 
areas the desired position cursor is, the variable rate cursor 12 to move to the left end of the reverse motion 
detector produces forward and reverse velocity control area 18 and produces, as in the other embodiment, maxi- 
signals that inform the video source interface 56 and the mum reverse velocity of the VTR or other video stor- 
variable rate icon generator 88 as to the user's desired age device. Similarly, maximum clockwise rotation of 
velocity. The video source interface 56 in turn produces 15 the control knob 81 causes the desired position cursor 
forward and reverse rate commands to control the ac- 12 to move to the right end of the forward motion area 
tual mass storage medium tape or other source. 20 and produces maximum forward velocity along the 

The variable rate icon generator 88 generates either a video sequence, 
reverse variable rate icon 89 or a forward variable rate The velocity of the video tape machine is propor- 
icon 90, depending on the direction and rate of desired 20 tional to the rotational distance of the active knob posi- 
motion. In the example shown in FIG. 5, the reverse tion indicator 79 into the forward or reverse regions 87 
variable rate icon 89 has grown larger toward the left to or 83, with maximum rotation corresponding to the 
fill part of the reverse motion area 18 in proportion to m a x i m um forward or reverse velocity of the video tape 
the amount of the maximum available reverse velocity machine: 
that is being used. In this example, the size of the reverse 

variable rate icon indicates that about 40% of the maxi- Dew 

mum available reverse velocity is being used. When Vcur ~ "zw" * Vmax 

forward motion is involved, the forward variable rate 

icon 90 operates in the same way, extending to the right 3Q where, 

to fill the forward motion area 20 in proportion to Vcur is the current velocity, 

amount of the maximum available forward velocity that Vmax is the maximum velocity, 

is being used Dcur is the current distance into the region, and 

As the current location along the active video se- I>max is the maximum distance into the region, 
quence changes, the detailed video pictorial timeline 11 35 The software of the human interface system can 
produced by the graphic interface generator 57 moves know the rotational position of the active knob position 
beneath the current position curser 13 to keep it posi- indicator 79 by any of several means. If the knob 81 
tioned over the current location. To accomplish this, employs a wire wound potentiometer, the output volt- 
the graphic interface generator 57 needs information age indicating its position can be translated into a digital 
indicative of the current position and the timecode ^ number by an analog-to-digital converter. Alterna- 
range, so that it can ascertain which frame samples (17 tively, the knob 81 can include an optical position en- 
in FIG. 3) it must retrieve from the local memory 60 in coder to provide a direct digital readout of its position, 
order to produce the correct detailed video pictorial The logic required to interpret the digital knob position 
timeline 11. It receives the current tape/source position information is similar to that described for cursor posi- 
information from the current tape/source position de- 45 tion detection in connection with FIGS. 4 and 5. 
tector 55 within the video source interface 56. Ideally, the locations on the knob 81 between the 

While the foregoing discussion concerning the opera- positional region 85 and the forward and reverse ra- 
tion of the desired position cursor 12 in the machine gions 83 and 87 are provided with detents to give the 
control window has referred to the motion of the cursor operator a slight amount of tactile feedback as the tran- 

12 into the forward and reverse motion areas 20 and 18, 50 sition is made from positional operation to velocity 
that motion need not actually occur. In one implemen- controlling operation. The knob 81 also has stops at the 
tation, when the mouse 70 is moved in a way that would outside ends of the reverse region 83 and forward re- 
cause the cursor 12 position to go beyond the edge of gion 87. 

detailed video pictorial timeline 11, the apparent posi- Another difference between the embodiment of the 
tion of the cursor 12 stops at that edge and yet the 55 machine control window shown in FIG. 6 and the one 
virtual motion of the cursor 12 can still be detected from shown in FIG. 5 is unrelated to the difference between 
additional real motion of the mouse 70. using a control knob 81 rather than a mouse 70, and 

FIG. 6 shows an alternative embodiment of the ma- could be used with either embodiment. The reverse 
chine control window input mechanism. In this embodi- variable rate icon 89' and forward variable rate icon 90' 
ment, the desired position cursor 12 is controlled by 60 in this implementation are both static, whereas the ones 
control knob 81 instead of a mouse or other cursor described above were dynamic. Whereas the icons de- 
control device 70. As before, when the desired position scribed above changed size, extending dynamically to 
cursor 12 is moved back and forth along the detailed indicate various velocities, the ones shown here do not 
video pictorial timeline 11, the current position cursor change, but nonetheless still suggest the idea of increas- 

13 follows and the frame selected is displayed in the 65 ing velocity by their sloped shape and the motion of the 
video window (14 in FIG. 1). This motion of the desired desired position cursor 12 along that shape, 
position cursor 12 within the detailed video pictorial One more difference between FIG. 6 and FIG. 5 is 
timeline 11 is controlled by the morion of the control that in the implementation shown in FIG. 6 the current 
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position cursor 13 is disposed above the detailed video processed 104 and it is determined whether a new 
pictorial timeline 11, rather than below it, as it was in source has been selected 105. 

FIGS. l f 4 and 5. In this implementation the current If a new source has been selected, 105- Yes, the new 
position cursor 13 remains visible even when it catches video is displayed 106 in the video window 14. A new 
up with the desired position cursor 12, whereas in the 5 main video pictorial timeline 10 and detailed video pic- 
other implementation, the two merge when they are tonal timeline 11 are created 107 and 108, as described 
ali S ned - in connection with FIG. 3. 

Referring now to FIGS. 7A and 7B, we see two Following these steps, as shown in FIG. 9B, a deter- 
simple timelines 21-A and 21-B extending horizontally mination is made 109 as to whether or not the video tape 
along a tune axis 24. and displaced vertically along a 10 machine or other video source is in motion. If no new 
layers axis 25 Tlie operator desires to tone align a point source had been selected, 105-No, this detennination 
of timehne 21-A with a point on timeline 21-B Using a 109 is ^ ^ ^ determination of st 105 . 

cursor 12 positioned on timeline 21-A near the location u me machine ^ m moti on, 109-Yes, the current posi- 
of what has now become the active time mark 27 the tion cursor ^ updated m ^ descr { he6 m connection 
operator has previously^ 15 ^ nG . 4, aid then a determination is made as to 

l*f£-f" T 5?" whether or not the operator is controlling the machine 
merly a 31 1 as the active time mark 27. When this m T *.,, u . . r . . . JZTz T t , 

design was performed, a vertical reference line 29 Si^fjfi? "* m 109 f 0 '^ d ff\' 

parallel to the layers axis 25 appeared at the location of " ? W f f?" * V^f" 15 

the active time mark 27. The operator then "dragged" 20 ^ g m * chm \ 1U 15 dLrectly, without updating 

the active timeline 21-A and the vertical reference line % C Z^ JT - ^ ir°' ^ u- 

29 to its present location using the mouse or other cur- * °F ™ " contr ? llm e the machine, Ill-Yes, 

sor control device while holding the control button th * deffl f ed Position cursor is updated 112 and a detenni- 

d 0WIL nation is made 113 as to whether or not the desired 

Since the reference line 29 now intersects timeline 25 p0si ^ on h b one of ^ rate areas (for- 

21-B at a point which is closer to the time mark 31 on ward motKm ^ or reverse motlon area >* If desired 
the left than it is to the one on the right or either end of P osltK,n curs ° r * in one of the variable rate areas, the 
the timeline 21-B, when the operator releases the con- deare d forward rate of motion or reverse rate of motion 
trol button on the mouse (70 in FIGS. 2 and 4), timeline IS detected < 86 m FIG. 5) and the detected rate of mo- 
21-A moves to the left along the time axis 24 to automat- 30 tion 18 converted to a command (56 in FIG. 5) to be sent 
ically and exactly align the active time mark 27 on time- 115 to VTR or other mass storage device (50 in 
line 21-A and the nearest time mark 31 on timeline 21-B. FIG. 2). If the cursor was not in the variable rate area, 
Note that for the purpose of time alignment, both ends H3-No, the desired forward rate of motion or reverse 
of the timelines 21 are also treated as if they were time rate of motio11 is detected (86 in FIG. 5) and the de- 
marks 31. 35 tected rate of motion is converted to a command (56 in 

In addition to aligning the timelines 21 and their time FIG. 5) to be sent 115 to the VTR or other mass storage 
marks 27 and 31 on the screen, the software also makes device (50 in FIG. 2). 

note of the timecodes and offsets that will be required to After the command is sent 115 to the mass storage 
access the under-lying video sources for recording or device, if path 111- Yes has been followed, or after it has 
previewing in a time-aligned manner. The various 40 °een deterrmned that the operator h not controlling the 
means used by video editing systems for physically machine, Ill-No, a determination is made 116 as to 
time-aligning video sequences, once the proper time- whether or not the operator has marked a selection to 
codes and offsets are known, is well known to those create a time mark, as described in connection with 
skilled in the video editing art. FIG. 4. If the operator has indicated a desire to create a 

FIG. 8 shows how video pictorial timelines 16 can be 45 tmie mark, 116- Yes, the mark is created 117 by the 
used in conjunction with the simple timelines 21 in the graphic interface generator 57 (FIG. 4) and placed in 
same Workspace 15 area, such as the layers area 32. storage in the local memory 60 as described in connec- 
This allows some of the video sequences to be seen with tion with FIG. 4 above. 

the extra visual cues provided by the video pictorial Whether or not the operator has marked a selection, 
timelines 16, while other video sequences are shown by 50 116-Yes or 116-No, as shown in FIG. 9C, a determina- 
the simple timelines 21 that save space and allow room tion is made 118 as to whether or not the operator has 
for more layers of video to be displayed. FIG. 8 also identified a time mark as the active time mark (27 in 
shows several inactive current position cursors 73. FIG. 7A). If not, the graphic timeline positions are 
These are left behind at the last position viewed, when converted 126 (FIG. 9D) to timecode values. If the 
the user's cursor 12 is detached from the current posi- 55 operator has designated a new active time mark, 118- 
tion cursor 13 by double clicking of the control button Yes, a further determination 119 is made as to whether 
on the mouse 70. this is the first time through this loop since the operator 

FIGS. 9A through 9D are a simplified logic flow identified the point. If this is the first pass through this 
diagram of a portion of the operator interface of the loop, 119-Yes, the closest mark on the same timeline is 
video editing system of the present invention. Referring 60 identified 120, the active time mark is high-lighted 121 
first to FIG. 9A, at start up the screen, the graphics and and a vertical reference line is created 121 through that 
the video window are initialized 100. The main loop of mark. Following these 119-Yes activities, or without 
the program then waits 101 for a frame sync pulse. performing these tasks if this was not the first time 
When the frame sync pulse is received, the graphics are through this loop, 119-No, operator input from the 
refreshed and a new frame of video is written 102 in the 65 mouse is detected and responded to 122. This input, if it 
video window. The status of the machine (video source) is present, is the desired horizontal motion of the time- 
is then polled and the current position timecode is line and the marks and vertical reference line associated 
stored 103. Keyboard and menu commands are then with it The response 122 is to move the timeline, marks 
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and vertical reference line according to the operator 
input 

Next, it is determined whether or not the switch on 
the mouse is still active 123. If the mouse button is still 
depressed, 123- Yes, the program advances to decision 5 
block 127,shown in FIG. 9D. If the mouse button has 
been released, 123-No, a determination is made 124 as to 
which mark on another timeline is closest to the vertical 
reference line associated with the active mark on this 
timeline. The selected timeline and its marks and verti- 10 
cal reference line are then moved 125 to be aligned with 
the closest mark on the other timeline, as shown in the 
transition from FIG. 7A to FIG. 7B. This new position 
information for the selected timeline is then converted 
126 to source timecode values. 15 

The user interface is then polled to determine 127 if 
the operator has selected either "record" or "preview" 
actions. If the operator has selected neither, 127-No, the 
program returns to the top of the main loop (FIG. 9A) 
and waits for the next frame sync pulse 101. If either 20 
"record" or "preview" were selected, 127- Yes, then 
commands are sent 123 to play all of the sources that 
were active in the Workspace 15 at the times corre- 
sponding to their locations in the Workspace, thus cre- 
ating the desired combined video sequence. If the se- 25 
lected operation is "preview" 129, the video and audio t 
are displayed only 130. If the selected operation is "re- 
cord" 131, the video and audio are recorded 132 to the 
master tape and the corresponding data is added to the 
edit decision list (EDL). In either event, the program 30 
returns to the top of the main loop and waits for the next 
frame sync pulse 101. 

While a preferred embodiment of the present inven- 
tion has been shown and described, it will be apparent 
to those skilled in the art that many changes and modifl- 35 
cations may be made without departing from the inven- 
tion in its broader aspects. For example, although the 
operator interface for a video editing system of the 
present invention, has been described as a single inter- 
face with several variations, portions of it could be 40 
modified for other uses and applied to those uses sepa- 
rately. The claims that follow are therefore intended to 
cover all such changes and modifications as fall within 
the true spirit and scope of the invention. 

I claim: 45 

1. A method for representing a video sequence, hav- 
ing a plurality of video frames, visually and spatially 
while minimizing local memory requirements, the 
method comprising the steps of: 

sampling each video frame of the video sequence to 50 
produce a frame sample for each video frame; 

storing the frame samples for the video frames in a 
local memory; and 

displaying the frame samples as a pictorial timeline 
representing an edge view of the video sequence. 55 

2. A method according to claim 1, wherein frame 
samples are displayed side by side and the sampling step 
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comprises the step of producing for each video frame a 
plurality of pixels that each represent an average video 
value within a corresponding region along a vertical 
axis of the video frame. 

3. A method according to claim 2, wherein the pro- 
ducing step comprises the step of emphasizing a dra- 
matic video feature within each region. 

4. A method according to claim 1; wherein the sam- 
pling step comprises the step of selecting a plurality of 
pixels along a vertical axis of each video frame. 

5. A method according to claim 4, wherein the verti- 
cal axis is taken through a center of each video frame. 

6. A method according to claim 1 further comprising 
the steps of: 

storing a corresponding video sequence in a mass 
storage medium; 

providing with each frame sample a pointer to the 
video frame from which the frame sample is pro- 
duced; 

selecting a location within the video sequence by 
indicating a frame sample from the pictorial time- 
line; and 

using the pointer from the indicated frame sample to 
access the video sequence at the selected location. 

7. A method according to claim 6 further comprising 
the steps of: 

responding to operator instructions by rearranging an 
order in which the frame samples are displayed; 
and 

outputting an edited video sequence according to the 
operator rearranged order of frame samples using 
the pointers to the video frames. 

8. A method according to claim 6 further comprising 
the step of providing a view into the stored video se- 
quence. 

9. A method according to claim 6 further comprising 
the steps of: 

sampling an audio signal to produce a plurality of 
audio samples which represent an envelope of the 
audio signal; 

storing the audio samples in the local memory; and 
displaying the audio samples together with the picto- 
rial timeline as a visual representation of the enve- 
lope over time. 

10. A method according to claim 9 further compris- 
ing the steps of: 

storing the audio signal in a mass storage medium; 

providing with each audio sample a pointer to a cor- 
responding portion of the audio signal in the mass 
storage medium; 

responding to operator instructions by rearranging an 
order in which the audio samples are displayed; 
and 

outputting an edited audio sequence according to the 
operator rearranged order of audio samples using 
the pointers to the audio signals. 

* * * * * 



03/11/2004, EAST 



Version: 1.4.1 



